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Abstract. The antibiotics time machine is an optimization question posed by Mira et a 1. 
on the design of antibiotic treatment plans to minimize antibiotic resistance. The problem 
is a variation of the Markov decision process. These authors asked if the problem can be 
solved efficiently. In this paper, we show that this problem is NP-hard in general. 


1. Problem statement and results 

Resistance to antibiotics in bacteria is a central problem in medicine. One method for 
mitigating its consequences is to rotate the use of antibiotics. A natural question arises: 
what is the optimal sequence of antibiotics one should apply? In this work, we follow the 
model proposed by Mira et a 1. |MCG + 14j . Suppose we have a population of bacteria, all 
of the same unmutated genotype (called the wild type). We are given a set of antibiotics. 
For each bacteria of a given type, an antibiotic mutates it to another type with some known 
probability. The problem is to find a sequence of antibiotics (called a treatment plan) 
that maximizes the fraction of bacteria returning to the wild type. This is equivalent to 
minimizing the fraction of antibiotics-induced mutations in the bacterial population. 

Here is a concrete description of the model. Let S' be a set of d states, where each state is 
a bacterial genotype. A d x d matrix T = (f^) is a transition matrix if t % j e [0,1] and each 
row sums to 1. Let T be a finite set consisting of K transition matrices, each corresponding 
to the effect of a given antibiotic on the genotypes of the bacterial population. That is, 
tij is the probability that a bacteria of genotype i will have genotype j after being treated 
with the corresponding antibiotic. The standard (d — l)-dimensional simplex is given 
by A d_1 = {(xi, x 2 , • • •, Xd) £ | Yli=i x i = 1 an d Xj > 0 for all i}. Let s = (1, 0,..., 0) £ 

A^ 1 be the starting state of the population, where all bacteria are of the first state (the wild 
type). Let N e N be the length of the treatment plan. The time machine TA i(d,K,N) 
is the solution to the following optimization problem 


maximize sTjT^ • • ■ Tjy s T 

subject to Tj,..., T n £ T. (1) 

The term time machine, coined by Mira et a 1., alludes to the idea of reversal to the wild 
type of unnecessary mutations done by antibiotic treatments. They proposed and solved a 
specific numerical instance of the time machine for 16 genotypes of the TEM /3-lactamase, 
with a specific model of mutation, and a set of up to six antibiotics. Their algorithm is to 
enumerate all possible treatment plans of a given length with the given set of antibiotics and 
output the optimal one. 

With K antibiotics, there are K N many treatment plans of length N , and thus direct 
enumeration is not an efficient algorithm. Mira et a 1. raised the question of whether an 
efficient algorithm exists. In this paper, we show that the general problem is NP-hard. 
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Theorem 1. The time machine optimization problem in © is NP-hard. 

We first consider an easier approximation problem: for a given threshold a G [0,1], decide 
whether there exists a treatment plan of length N such that 

sTxT 2 • • • T]\fS ^ Y a 

subject to Tj,..., TV G T. (2) 

Clearly if one can solve ([Tj) in polynomial time, then one can also solve (J2J) in polynomial 
time. Our main result states that the latter problem is NP-hard, and thus Theorem [1] follows 
as a corollary. 

Theorem 2. The time machine decision problem in ([2]) is NP-hard. 

Our proof relies on reducing 3-SAT to a special instance of (J2J) where a — 1. Since 3-SAT 
is NP-hard |Kar72j . we have that ([2]) is also. The reduction construction is described in 
Section [2l The proof that this is a correct reduction is in Section [31 

1.1. Related literature. The time machine may also arise in large scale robotics control, 
where one has a large number of robots with d possible states, and each Tj is an instruction 
which changes each robot’s state independently at random. In general, it can be viewed 
as a Markov decision problem with infinitely many states. Indeed, if there was only one 
bacteria instead of a population, one may apply an antibiotic Tj, check the new state of the 
bacteria, and repeat. In this case, the time machine is a Markov decision process on d states, 
which can be solved by dynamic programming. When there is a population of bacteria, the 
population state is a point in the standard ( d — l)-dimensional simplex A d_1 , which is an 
infinite set. 

There is a large literature on Markov decision processes, see [FSA02] and references therein. 
However, we are unaware of any existing results on the time machine, or the equivalent of 
Theorem Q] Closely related to the time machine is the partially observable Markov decision 
process [DST3] . where the underlying system is assumed to be in one of the unobservable d 
states, and s G A d reflects the uncertainty over the true state. It would be interesting to see 
if techniques from this setup can yield polynomial time approximations to the time machine. 

1.2. Open problems. Many interesting questions remain on the antibiotics time machine. 
We mention two examples. The last question was raised by Joe Kileel. 

(i) For what classes of antibiotics T is the problem solvable in polynomial time? Are 
there biologically relevant instances? 

(ii) When does the limit linijv-joo nrax^gr sTjT 2 ■ ■ ■ T/yS T exist? If so, how fast is the 
convergence? In this case, for large N, can one produce an approximate time machine 
in polynomial time? 


2. Reduction construction 

Let X = {xi,..., x n } be a set of Boolean variables taking Boolean values {+,—}• A triple 
(£iXi,EjXj,£kXk) with £i,£j,£k G {+, —} is called a clause. Let C = {ci,...,c m } be a set 
of clauses. Write (xj = Vi )j to mean X* = Uj for i — 1,..., n. An assignment (xj = v\ )j 
satisfies a clause c = (£iXi,£jXj,EkXk) if (£i,£j,£k) ^ (vi,Vj,Vk)- Without loss of generality, 
assume each x G X actually occurs (negated or not) somewhere. The 3-SAT problem is: 
given a set of variables and clauses, decide whether there exists an assignment that satisfies 
all clauses. In this section, we reduce a 3-SAT instance to an instance of (121) . 
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Let N = m + 2 and a — 1. Create d = 3n + m + 3 states and K = 7m + 2 transition 
matrices. Create the start state s, a “death” state d, and a “tally” state f. For each variable 
x G X, create states x, x~,x + . For each clause c = (EiXi,EjXj,EkXk) G C, create state c. We 
identify each state with its characteristic (row) vector and define each transition matrix by 
its action on these basis vectors and extend linearly. 

For each choice of (vi,Vj,Vk) G {+, — } 3 \ {(£$, Ej, £*,)}, create a transition matrix T = 
ag f 0 u 0WS: 


z T 


"d z = s 
f z = c 

< x. v / z = x.£, l G {i, j, k} 
d z = xJ Vi , £ G {i,j, k} 
k z otherwise. 


Dehne a starting matrix S' by 


z S 


P (ExGA- X + Ecec C ) 

d 


z = s 

otherwise, 


where p = 1/ (n + m ). Lastly, dehne a final matrix F by 

fs z G W : Vi G {+, -}} 
zF = < s f 

Id otherwise. 

Let T consist of S, F, and the 7m transition matrices Tc" V3 ' Vk dehned above. This concludes 
the reduction construction. 


3. Proof of Main Theorem 

We shall prove that with the setup in Section [21 the time machine decision problem ([2D 
solves the 3-SAT problem with the given clauses. Since the latter is NP-hard, this implies 
that (J2D is also. 

We refer to the fraction of the bacterial population in a given state as its weight. First, 
suppose {xi = Vi)i is a satisfying assignment. Apply S to s. For each c = ( EiXi,£jXj,EkXk ) G 
C, apply Tc l ’ Vj ’ Vk , which exists as c is satisfied. Finally, apply F. 

Lemma 3. The sequence of matrices given above sends all weights back to s. That is, it is 
a solution of (J2D for a = 1. 

Proof. For each x G X, the matrix S sends some weight to x. The weight is moved to x Vi 
the first time ±x occurs in a clause c, and is moved back to s by F at the end. For each 
c G C, the matrix S sends some weight to c. This weight is moved to f by Tf x ’ Vj,Vk ^ anc ( i s 
moved back to s by F at the end. Therefore all the weights returns to s, as desired. □ 

Conversely, suppose there is a sequence T), T 2 ,... ,T N of transition matrices so that 

sTiT 2 • • • TjvS T = 1. 

The aim is to extract a satisfying assignment. 

Consider the process of applying the transition matrices T t to s sequentially in N steps. 
Note that any weight at d stays at d forever. As such, to achieve full weight at s after N 





Tran-Yang 4 


May 12, 2015 


steps, the state d cannot receive weight at any point in the processEI Consequently, it is 
clear that the first matrix to apply has to be S, as we have T(s) = d for T € T \ 

The only way for s to gain weight is to apply F. Prior to applying F for the first time, 
all weights must be supported on the xf and f, as to avoid losing any weight to the death 
state d. In particular, for each clause c = (£iXi,£jXj,£kXk) G C, state c must no longer carry 
weight at this point. That is, after c receives some weight by S in the first step, it must 
subsequently lose the weight, which can only be achieved by applying an associated matrix 
jiv t ,vj,v k some choice of (vi, v v u fc ). This takes (at least) one step for each clause. Since the 
sequence of matrices is of length precisely N = m + 2, we conclude that the sequence starts 
with S, finishes with F, and contains precisely one matrix corresponding to each clause. 

When S is applied, the full weight at s is split into n + m packets, each of weight 
p — 1 /(n + m). Since no other matrices split weights, we may consider the remaining process 
as a discrete system moving each packet as a unit. We already saw that the m packets 
associated to the clauses are moved to f during the middle m steps. It remains to analyze 
the remaining n packets associated to the variables. To avoid moving any weight to the 
death state d, the packet at x, : can only be moved to xf or x~. Once this happens, the 
packet can only be moved again by F at the last step. So at the penultimate step, there is 
a packet on x$ for exactly a choice of fy for each i. The following lemma finishes the proof. 

Lemma 4. For each i, let v t be such that xff has nonzero weight after m + 1 steps, i.e. , 

sTiT 2 • • ■ T/ V _ 1 (xf i ) T > 0. 

Then (xi = vf)i is a satisfying assignment. 

Indeed, consider a clause c = (eiXi, £jXj, £kXk ) G C. We know that (exactly) one associated 
transition matrix Tf'" V:i ’ Vk is used. Suppose, towards a contradiction, that v ( $ V£ for some 
£ G {i,j, k}. After applying Tf z ’ Vj ’ Vk , the packet corresponding to xi is at xf or d, with no way 
of moving to xf, a contradiction. So (7$ Vj, vf) = (■ Vi,Vj,Vk ) ^ ( £i,£j,£k ) by construction, 
implying that clause c is satisfied. □ 

This shows that a 3-SAT instance has a solution if and only if the associated time machine 
can attain a threshold of 1. We therefore conclude that the time machine decision problem 
is NP-hard, as desired. 
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